Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This fixes a performance problem with (at least) incremental backups. My incr backups were slower the longer it was since the last reboot. Investigation showed the delay was proportional to the size of open_files.dat, with the indexing phase taking over an hour a week after a reboot instead of ~4 minutes seen immediately after a reboot. The problem is that open_files.dat, and the in-memory representation in 'open_write_files', contains references to many files which are no longer open. On my system, there should be around 300 open files, but after a week, we think there are about 90K open files.
Code in ChangeJournalWatcher.cpp failed to remove many files from open_write_files. Most of that was because USN_REASON_RENAME_OLD_NAME should be treated like USN_REASON_CLOSE when handling a normal file, removing the file from open_write_files.
The other leak was a timing problem - if multiple USN change journal records for a single file are split across multiple calls to ChangeJournalWatcher::update(), the file might get added to open_write_files after the first update() call, but the USN records for the file in the second update() could be handled completely with local_open_write_files, not noticing that the file had already been added to open_write_files and should be removed on a close journal record. Fix is, when adding a file to local_open_write_files, check whether the file is already in open_write_files, and if yes, remove it from both versions of the map when encountering a close journal record.
Including a compile minor changes. First, PrettyPrintTime was returning an empty string for times less than a second. Second, ChangeJournalWatcher::update_longliving should run its loops within a BeginTransaction/EndTransaction so the updates to the mfiles table are batched.